Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert PyPSA Network object to Grid object and profiles #652

Merged
merged 19 commits into from
Nov 10, 2022
Merged

Convert PyPSA Network object to Grid object and profiles #652

merged 19 commits into from
Nov 10, 2022

Conversation

rouille
Copy link
Collaborator

@rouille rouille commented Jun 9, 2022 •
edited
Loading

Pull Request doc

Purpose

Convert a PyPSA Network object to a Grid object and times series for demand, hydro, solar and wind

What the code is doing

Use existing code in #608 to convert a in memory PyPSA Network object into objects that can be handled by the scenario framework. The feature is incomplete:

  • storage is not yet implemented
  • plant, branch, etc data frames of the Grid object have missing/new columns and different types than what is expected (see powersimdata.input.const module)
  • hydro and wind profiles cannot be extracted from a PyPSA-Eur network

For example:

Reading a PyPSA-Eur netCDF file from Zenodo:

>>> from powersimdata.network.europe_tub.model import TUB
>>> from powersimdata.input.converter.pypsa_to_grid import FromPyPSA
>>> tub = TUB("Europe", reduction=512)
Title: PyPSA-Eur: An Open Optimisation Model of the European Transmission System (Dataset)
Keywords: power system model, capacity expansion model, energy system model, electricity system model, python
Publication date: 2021-09-22
DOI: 10.5281/zenodo.5521712
Total size: 1767.1 MB

Link: https://zenodo.org/api/files/6e58c6dc-083b-46c8-8255-8516e3eb0ed8/networks.zip   size: 1767.1 MB
networks.zip is already downloaded correctly.
All files have been downloaded.
>>> tub.build()
WARNING:pypsa.io:
Importing PyPSA from older version of PyPSA than current version.
Please read the release notes at https://pypsa.readthedocs.io/en/latest/release_notes.html
carefully to prepare your network for import.
Currently used PyPSA version [0, 19, 3], imported network file PyPSA version [0, 18, 0].

INFO:pypsa.io:Imported network elec_s_512.nc has buses, carriers, generators, lines, links, loads, storage_units
>>> grid = FromPyPSA(tub.network)
/Users/brdo/CEM/PowerSimData/powersimdata/input/converter/pypsa_to_grid.py:195: UserWarning: Substations could not be parsed.
  warnings.warn("Substations could not be parsed.")
/Users/brdo/CEM/PowerSimData/powersimdata/input/converter/pypsa_to_grid.py:247: UserWarning: The export of storages are not implemented yet.
  warnings.warn("The export of storages are not implemented yet.")
>>> grid.plant
                       weight        Pmax bus_id        type  Pmin  ramp_30  status
plant_id                                                                           
AL0 0 offwind-ac   698.367965    0.000000  AL0 0  offwind-ac   0.0      0.0     NaN
AL0 0 onwind      1762.449543    0.000000  AL0 0      onwind   0.0      0.0     NaN
AL0 0 solar       2347.489775    0.000000  AL0 0       solar   0.0      0.0     NaN
AT0 0 onwind       171.690009    0.000000  AT0 0      onwind   0.0      0.0     NaN
AT0 0 ror          297.580368  297.580368  AT0 0         ror   0.0      0.0     NaN
...                       ...         ...    ...         ...   ...      ...     ...
SK0 2 ror          188.960000  188.960000  SK0 2         ror   0.0      0.0     NaN
SK0 2 solar        993.396581    0.000000  SK0 2       solar   0.0      0.0     NaN
SK0 3 onwind       642.421167    0.000000  SK0 3      onwind   0.0      0.0     NaN
SK0 3 ror           27.850000   27.850000  SK0 3         ror   0.0      0.0     NaN
SK0 3 solar        501.830654    0.000000  SK0 3       solar   0.0      0.0     NaN

[2305 rows x 7 columns]

Extracting demand and solar profiles work well:

>>> from powersimdata.input.converter.pypsa_to_profiles import (
...     get_pypsa_demand_profile,
...     get_pypsa_gen_profile,
... )
>>> demand = get_pypsa_demand_profile(tub.network)
>>> demand
                         AL0 0       AT0 0       AT0 1       AT0 2       AT0 3   ...      SI0 2       SK0 0       SK0 1        SK0 2       SK0 3 
UTC                                                                              ...                                                             
2013-01-01 00:00:00  683.702703  537.098973  695.257525  325.506959  570.104515  ...  206.150601  600.840434  699.647401  1142.382463  342.129702
2013-01-01 01:00:00  634.945946  514.860245  666.470200  312.029256  546.499183  ...  197.094890  573.441247  667.742474  1090.288182  326.528096
2013-01-01 02:00:00  593.391892  487.314548  630.813172  295.335282  517.260762  ...  193.543630  554.671726  645.886344  1054.601549  315.840382
2013-01-01 03:00:00  554.054054  485.545558  628.523271  294.263192  515.383065  ...  190.702623  551.219860  641.866825  1048.038490  313.874826
2013-01-01 04:00:00  555.162162  491.694904  636.483402  297.989981  521.910297  ...  192.300689  552.082826  642.871705  1049.679254  314.366215
...                         ...         ...         ...         ...         ...  ...         ...         ...         ...          ...         ...
2013-12-31 19:00:00  710.297297  625.211508  809.316398  378.907254  663.631698  ...  217.514630  638.163736  743.108442  1213.345539  363.382283
2013-12-31 20:00:00  665.418919  594.549020  769.624782  360.324360  631.084954  ...  199.758334  615.079381  716.227913  1169.455082  350.237623
2013-12-31 21:00:00  683.148649  619.651826  802.119566  375.537828  657.730365  ...  189.282119  608.607133  708.691317  1157.149346  346.552205
2013-12-31 22:00:00  690.351351  588.483912  761.773694  356.648623  624.647136  ...  180.226408  598.683018  697.135202  1138.280551  340.901229
2013-12-31 23:00:00  671.513514  568.098412  735.385312  344.294061  603.008916  ...  192.300689  586.385745  682.815668  1114.899654  333.898934

[8760 rows x 509 columns]
>>> solar = get_pypsa_gen_profile(tub.network, "solar")
>>> solar
                     AL0 0 solar  AT0 0 solar  AT0 1 solar  AT0 2 solar  AT0 3 solar  ...  SI0 2 solar  SK0 0 solar  SK0 1 solar  SK0 2 solar  SK0 3 solar
UTC                                                                                   ...                                                                 
2013-01-01 00:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 01:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 02:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 03:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 04:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
...                          ...          ...          ...          ...          ...  ...          ...          ...          ...          ...          ...
2013-12-31 19:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 20:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 21:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 22:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 23:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0

[8760 rows x 502 columns]

But not wind and hydro:

>>> wind = get_pypsa_gen_profile(tub.network, "wind")
>>> wind
Empty DataFrame
Columns: []
Index: [2013-01-01 00:00:00, 2013-01-01 01:00:00, 2013-01-01 02:00:00, 2013-01-01 03:00:00, 2013-01-01 04:00:00, 2013-01-01 05:00:00, 2013-01-01 06:00:00, 2013-01-01 07:00:00, 2013-01-01 08:00:00, 2013-01-01 09:00:00, 2013-01-01 10:00:00, 2013-01-01 11:00:00, 2013-01-01 12:00:00, 2013-01-01 13:00:00, 2013-01-01 14:00:00, 2013-01-01 15:00:00, 2013-01-01 16:00:00, 2013-01-01 17:00:00, 2013-01-01 18:00:00, 2013-01-01 19:00:00, 2013-01-01 20:00:00, 2013-01-01 21:00:00, 2013-01-01 22:00:00, 2013-01-01 23:00:00, 2013-01-02 00:00:00, 2013-01-02 01:00:00, 2013-01-02 02:00:00, 2013-01-02 03:00:00, 2013-01-02 04:00:00, 2013-01-02 05:00:00, 2013-01-02 06:00:00, 2013-01-02 07:00:00, 2013-01-02 08:00:00, 2013-01-02 09:00:00, 2013-01-02 10:00:00, 2013-01-02 11:00:00, 2013-01-02 12:00:00, 2013-01-02 13:00:00, 2013-01-02 14:00:00, 2013-01-02 15:00:00, 2013-01-02 16:00:00, 2013-01-02 17:00:00, 2013-01-02 18:00:00, 2013-01-02 19:00:00, 2013-01-02 20:00:00, 2013-01-02 21:00:00, 2013-01-02 22:00:00, 2013-01-02 23:00:00, 2013-01-03 00:00:00, 2013-01-03 01:00:00, 2013-01-03 02:00:00, 2013-01-03 03:00:00, 2013-01-03 04:00:00, 2013-01-03 05:00:00, 2013-01-03 06:00:00, 2013-01-03 07:00:00, 2013-01-03 08:00:00, 2013-01-03 09:00:00, 2013-01-03 10:00:00, 2013-01-03 11:00:00, 2013-01-03 12:00:00, 2013-01-03 13:00:00, 2013-01-03 14:00:00, 2013-01-03 15:00:00, 2013-01-03 16:00:00, 2013-01-03 17:00:00, 2013-01-03 18:00:00, 2013-01-03 19:00:00, 2013-01-03 20:00:00, 2013-01-03 21:00:00, 2013-01-03 22:00:00, 2013-01-03 23:00:00, 2013-01-04 00:00:00, 2013-01-04 01:00:00, 2013-01-04 02:00:00, 2013-01-04 03:00:00, 2013-01-04 04:00:00, 2013-01-04 05:00:00, 2013-01-04 06:00:00, 2013-01-04 07:00:00, 2013-01-04 08:00:00, 2013-01-04 09:00:00, 2013-01-04 10:00:00, 2013-01-04 11:00:00, 2013-01-04 12:00:00, 2013-01-04 13:00:00, 2013-01-04 14:00:00, 2013-01-04 15:00:00, 2013-01-04 16:00:00, 2013-01-04 17:00:00, 2013-01-04 18:00:00, 2013-01-04 19:00:00, 2013-01-04 20:00:00, 2013-01-04 21:00:00, 2013-01-04 22:00:00, 2013-01-04 23:00:00, 2013-01-05 00:00:00, 2013-01-05 01:00:00, 2013-01-05 02:00:00, 2013-01-05 03:00:00, ...]

[8760 rows x 0 columns]
>>> hydro = get_pypsa_gen_profile(tub.network, "hydro")
>>> hydro
Empty DataFrame
Columns: []
Index: [2013-01-01 00:00:00, 2013-01-01 01:00:00, 2013-01-01 02:00:00, 2013-01-01 03:00:00, 2013-01-01 04:00:00, 2013-01-01 05:00:00, 2013-01-01 06:00:00, 2013-01-01 07:00:00, 2013-01-01 08:00:00, 2013-01-01 09:00:00, 2013-01-01 10:00:00, 2013-01-01 11:00:00, 2013-01-01 12:00:00, 2013-01-01 13:00:00, 2013-01-01 14:00:00, 2013-01-01 15:00:00, 2013-01-01 16:00:00, 2013-01-01 17:00:00, 2013-01-01 18:00:00, 2013-01-01 19:00:00, 2013-01-01 20:00:00, 2013-01-01 21:00:00, 2013-01-01 22:00:00, 2013-01-01 23:00:00, 2013-01-02 00:00:00, 2013-01-02 01:00:00, 2013-01-02 02:00:00, 2013-01-02 03:00:00, 2013-01-02 04:00:00, 2013-01-02 05:00:00, 2013-01-02 06:00:00, 2013-01-02 07:00:00, 2013-01-02 08:00:00, 2013-01-02 09:00:00, 2013-01-02 10:00:00, 2013-01-02 11:00:00, 2013-01-02 12:00:00, 2013-01-02 13:00:00, 2013-01-02 14:00:00, 2013-01-02 15:00:00, 2013-01-02 16:00:00, 2013-01-02 17:00:00, 2013-01-02 18:00:00, 2013-01-02 19:00:00, 2013-01-02 20:00:00, 2013-01-02 21:00:00, 2013-01-02 22:00:00, 2013-01-02 23:00:00, 2013-01-03 00:00:00, 2013-01-03 01:00:00, 2013-01-03 02:00:00, 2013-01-03 03:00:00, 2013-01-03 04:00:00, 2013-01-03 05:00:00, 2013-01-03 06:00:00, 2013-01-03 07:00:00, 2013-01-03 08:00:00, 2013-01-03 09:00:00, 2013-01-03 10:00:00, 2013-01-03 11:00:00, 2013-01-03 12:00:00, 2013-01-03 13:00:00, 2013-01-03 14:00:00, 2013-01-03 15:00:00, 2013-01-03 16:00:00, 2013-01-03 17:00:00, 2013-01-03 18:00:00, 2013-01-03 19:00:00, 2013-01-03 20:00:00, 2013-01-03 21:00:00, 2013-01-03 22:00:00, 2013-01-03 23:00:00, 2013-01-04 00:00:00, 2013-01-04 01:00:00, 2013-01-04 02:00:00, 2013-01-04 03:00:00, 2013-01-04 04:00:00, 2013-01-04 05:00:00, 2013-01-04 06:00:00, 2013-01-04 07:00:00, 2013-01-04 08:00:00, 2013-01-04 09:00:00, 2013-01-04 10:00:00, 2013-01-04 11:00:00, 2013-01-04 12:00:00, 2013-01-04 13:00:00, 2013-01-04 14:00:00, 2013-01-04 15:00:00, 2013-01-04 16:00:00, 2013-01-04 17:00:00, 2013-01-04 18:00:00, 2013-01-04 19:00:00, 2013-01-04 20:00:00, 2013-01-04 21:00:00, 2013-01-04 22:00:00, 2013-01-04 23:00:00, 2013-01-05 00:00:00, 2013-01-05 01:00:00, 2013-01-05 02:00:00, 2013-01-05 03:00:00, ...]

[8760 rows x 0 columns]

Testing

Add new unit tests

Where to look

Two new modules:

  • pypsa_to_grid that creates a Grid object from the Network object
  • pypsa_to_profiles that extracts the profiles from the Network object

Usage Example/Visuals

How the code can be used and/or images of any graphs, tables or other visuals (not always applicable).

Time estimate

How long will it take for reviewers and observers to understand this code change?

@rouille rouille self-assigned this Jun 9, 2022
@rouille rouille force-pushed the ben/import branch 2 times, most recently from 00e898b to a32235e Compare June 16, 2022 19:26
@rouille rouille force-pushed the ben/import branch 2 times, most recently from 46894c5 to c0d9fbf Compare September 16, 2022 18:47
@rouille rouille force-pushed the ben/import branch 3 times, most recently from eba5dfd to 2f2a445 Compare October 27, 2022 20:45
rouille and others added 10 commits November 10, 2022 13:22
@rouille
fix: enable grid equality for back converted pypsa networks (#689)
9707a82
@rouille
refactor: create library of constants for grid object, casemat file a…
3e103fc 
…nd pypsa translators (#667)
@rouille
feat: normalize inflow profiles by max
9a4d290
@rouille
Merge pull request #675 from Breakthrough-Energy/ben/grideq
16b190d 
Enable roundtrip conversion PyPSA Network <--> PowerSimData Grid objects
@rouille
refactor: simplify logic
0444b31
@FabianHofmann @rouille
feat: extract substation from arbitrary pypsa networks (#674)
b805325
@rouille
refactor: add inflow to column name of carriers with inflow profiles
adf9309
@FabianHofmann @rouille
feat: support hydro inflow functionality (#691)
04f2d4d
@rouille
Merge pull request #682 from Breakthrough-Energy/ben/profile
0c7d0d5 
Extract profiles from PyPSA's Network objects
@FabianHofmann @rouille
fix: add geographical coordinates to branch and plant data frames and…
3d17f0f 
… fix bus assignment/naming (#703)
@rouille rouille marked this pull request as ready for review November 10, 2022 21:22
@rouille
Copy link
Collaborator Author

rouille commented Nov 10, 2022

@BainanXia, @jenhagg, this is ready to be merged into develop. We can successfully convert a PyPSA Network object to a Grid object and extract profiles. The next step is to set the model immutables after conversion (or during, to be discussed) but this can be done off the develop branch with potential refactoring of the TUB and Grid class.

BainanXia
BainanXia approved these changes Nov 10, 2022
View reviewed changes
Copy link
Collaborator

@BainanXia BainanXia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! Thanks 🚀

@rouille rouille merged commit 1614d2e into develop Nov 10, 2022
@rouille rouille deleted the ben/import branch November 10, 2022 22:40
@jenhagg jenhagg mentioned this pull request Dec 10, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jenhagg jenhagg jenhagg left review comments

@BainanXia BainanXia BainanXia approved these changes

@merrielle merrielle Awaiting requested review from merrielle

Assignees

@rouille rouille

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@rouille @BainanXia @jenhagg @chrstphtrs @FabianHofmann